Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR
For the newly failing unit tests (#2315), the problem was triggered by assemble_fibermap in PR #2313 needing CalibFinder to look up which fibers need FIBERSTATUS bits set, but
fallback_on_dark_not_found=True
. I considered updating CalibFinder itself to only crash if a missing dark was requested (instead of crashing at initialization time whether or not the dark was needed), but that was going to be a bigger refactor andfallback_on_dark_not_found=True
within assemble_fibermap worked fine.For bootcalib #2030, the unit tests now download from https://data.desi.lbl.gov/public/epo/example_files/spectest instead of https://portal.nersc.gov/project/desi/data/spectest which no longer exists. Full disclosure: previously the tests were run out of the desispec clone directory and the tests files were left behind in the top-level dir, and re-used if they already existed from a previous test run. Now the test is using a temp directory every time, but that means it also has to download the files every time. If the download fails it just skips the test (which is what is has been doing for over a year since the old URL disappeared anyway). When running at NERSC, the download is a fraction of a second, but we should keep an eye on this if it becomes problematically slow from elsewhere.
In general, I cleaned up the other tests so that they don't write test files into the desispec git clone directory. In addition to being good practice, this now enables tests to be run from a GPU compute node which mounts /global/common/software readonly. To minimize changes to the tests themselves, the basic model I followed was:
origdir=os.getcwd()
, creates a temporary directorytestdir=tempfile.mkdtemp()
, and move there withos.chdir(testdir)
.os.chdir(self.testdir)
in case any previous test had moved out of that diros.chdir(origdir)
so that the next test suite starts from a known location.Some tests already had some usage of tempfile, e.g. to specify the full path of every file instead of
os.chdir
+ writing files without a path. I left those as-is.Summarizing, this now works to run tests from a GPU compute node that has mounted a git clone directory in read-only mode: